1. Technical Field
The present invention relates in general to cluster multiprocessing systems and in particular to dynamic reconfiguration of cluster multiprocessing systems. Still more particularly, the present invention relates to dynamic reconfiguration of highly available cluster multiprocessing systems.
2. Description of the Related Art
High availability (HA) is gaining widespread commercial acceptance as an alternative to fault tolerance for mission-critical computing platforms. Fault tolerant data processing systems rely on specialized hardware to detect hardware faults and switch to a redundant hardware component, regardless of whether the component is a processor, memory board, hard disk drive, adapter, power supply, etc. While providing seamless cutover and uninterrupted performance, fault tolerant systems are expensive due to the redundant hardware requirement. Additionally, fault tolerant systems do not address software errors, a more common source of data processing system failure.
High availability utilizes standard hardware, but provides software allowing resources to be shared system wide. When a node, component, or application fails, an alternative path to the desired resource is quickly established. The brief interruption required to reestablish availability of the resource is acceptable in many situations. The hardware costs are significantly less than fault tolerant systems, and backup facilities may be utilized during normal operation.
Highly available systems are often implemented as clustered multiprocessor (CMP) systems. A cluster includes a plurality of nodes or processors connected to shared resources, such as shared external hard disks. Typically, each node runs a server or "back end" application permitting access to the shared resources. A node may "own" a set of resources--disks, volume groups, file systems, networks, networks addresses and/or applications--as long as that node is available. When that node goes down, access to the resources is provided through a different node.
An active configuration comprises a set of hardware and software entities together with a set of relationships among these entities, the combination of entities and relationships delivering services to users. Hardware entities specify nodes, adapters, shared disks, etc. while software entities specify failover and reintegration policies. For example, a particular software entity may specify that an application server should failover to node B when node A fails. It may also specify whether the application server should fail back to node A when node A reintegrates.
Within clustered multiprocessing systems, it would be advantageous to be able to dynamically reconfigure an active cluster, changing the cluster configuration without having to stop and then restart cluster services. Dynamic change of an active configuration preferably changes the entity and relationship sets while the system is running, with as little disruption of service as possible. Such dynamic configuration changes are required when the cluster is serving mission-critical applications that cannot be brought down for long periods of time (and preferably are not brought down at all).
An example of a situation requiring persistent support for dynamic configuration changes is performing a hardware upgrade within a four node cluster (nodes A, B, C and D). A user may need to bring down the node to be upgraded, such as node D, upgrade the hardware, rejoin node D to the cluster, and possibly make configuration changes. If node D were equipped with a faster processor and/or additional memory, for instance, the user may wish node D to become the primary system for an application server previously run on a different node. The user will desire to make these changes and will want the changes to be preserved across power outages and cluster reboots.
Another example of a situation requiring dynamic configuration changes involves transient dynamic configuration changes. If the workload of a node temporarily increases, the user may wish to move an application server previously run on that system to another node. Since the increased workload is not normal, the change need not be preserved across cluster reboots.
At least one prior art cluster software package--HACMP for AIX.RTM., available from International Business Machines Corporation of Armonk, N.Y.--provides some dynamic reconfiguration capabilities. Each node includes a default configuration which is copied into the active configuration for the respective node at cluster start-up. The default configuration may be modified while the cluster is active and copied into the default configurations of other nodes. This modified default configuration is then copied into a staging configuration in each active node. The new configuration is verified and, when the daemons for each cluster node are refreshed, copied into the active configuration for active nodes. Cluster services for an inactive nodes added by the reconfiguration may then be started.
The prior art system for dynamic reconfiguration has several limitations. First, multiple reconfigurations cannot be synchronized. When a second reconfiguration is initiated while a dynamic reconfiguration is in progress, the presence of a staging configuration on any cluster node acts as a lock preventing initiation of a new dynamic reconfiguration event.
Second, the prior art system cannot be utilized to effect dynamic changes when multiple software components are involved in applying different parts of the changes to the configuration. Where a dynamic configuration change involving multiple software components fails, the changes already performed up to the point of failure must be rolled back. This is much more complex than dynamically changing only a single component, and reverting to an old configuration if the attempted configuration change fails. Thus, the changes which may be made dynamically are limited.
It would be desirable, therefore, to provide an cluster multiprocessing system with support for dynamic changes involving multiple software components, and for synchronizing multiple dynamic reconfigurations. It would further be desirable to coordinate dynamic configuration changes with other events in a system and to make the dynamic changes in a fail safe manner.